Motion information display apparatus and method

ABSTRACT

A motion information display apparatus according to an embodiment includes obtaining circuitry, identifying circuitry, and display controlling circuitry. The obtaining circuitry obtains a plurality of pieces of moving image information, and motion information that indicates a motion of a subject included in each of the moving image information. The identifying circuitry identifies a frame corresponding to a timing of a predetermined motion, from each frame group included in each of the moving image information, based on the motion information. The display controlling circuitry performs display control of the plurality of pieces of moving image information, using the frame corresponding to a timing of a predetermined motion.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a continuation of PCT international application Ser.No. PCT/JP2014/051140 filed on Jan. 21, 2014 which designates the UnitedStates, incorporated herein by reference, and which claims the benefitof priority from Japanese Patent Application No. 2013-008605, filed onJan. 21, 2013, the entire contents of which are incorporated herein byreference.

FIELD

Embodiments described herein relate generally to a motion informationdisplay apparatus and a method.

BACKGROUND

Conventionally, cooperative assistance by many specialists is performedin rehabilitation for the purpose of making a person's life better, theperson being with mental and physical disabilities or a congenitaldisorder due to various causes such as a disease, an injury, or an agingprocess. For example, in the rehabilitation, cooperative assistance isperformed by many specialists including a rehabilitation medicalspecialist, a rehabilitation nurse, a physical therapist, anoccupational therapist, a speech-language-hearing therapist, a clinicalpsychologist, a prosthetist, and a social worker.

Meanwhile, in recent years, motion capture technologies that digitallyrecord a movement of a person or an object have been developed. Assystems of the motion capture technologies, an optical system, amechanical system, a magnetic system, and a camera system are known. Asan example, a camera system is known, which digitally record a movementof a person by causing the person to wear markers, detecting the markerswith a tracker such as a camera, and processing the detected markers.Further, as a system without using the markers and the tracker, a systemusing an infrared light sensor is known, which digitally records amovement of a person by measuring a distance from the sensor to theperson, and detecting various movements of the size or a skeleton of theperson. As the sensor using such a system, Kinect (registered trademark)is known, for example.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a block diagram illustrating a configuration diagram of amedical-use information system according to a first embodiment;

FIG. 2 is a block diagram illustrating a detailed configuration exampleof motion information collecting circuitry according to the firstembodiment;

FIG. 3A is a diagram for describing processing of motion informationgenerating circuitry according to the first embodiment;

FIG. 3B is a diagram for describing the processing of the motioninformation generating circuitry according to the first embodiment;

FIG. 3C is a diagram for describing the processing of the motioninformation generating circuitry according to the first embodiment;

FIG. 4 is a diagram illustrating an example of skeleton informationgenerated by the motion information generating circuitry according tothe first embodiment;

FIG. 5 is a block diagram illustrating a detailed configuration exampleof a motion information display apparatus according to the firstembodiment;

FIG. 6A is a diagram for describing processing of identifying circuitryaccording to the first embodiment;

FIG. 6B is a diagram for describing the processing of the identifyingcircuitry according to the first embodiment;

FIG. 7 is a diagram for describing processing of display controllingcircuitry according to the first embodiment;

FIG. 8 is a flowchart for describing an example of a processingprocedure of the motion information display apparatus according to thefirst embodiment;

FIG. 9A is a diagram for describing processing of identifying circuitryaccording to a second embodiment;

FIG. 9B is a diagram for describing the processing of the identifyingcircuitry according to the second embodiment;

FIG. 10 is a diagram for describing processing of display controllingcircuitry according to the second embodiment;

FIG. 11 is a flowchart for describing an example of a processingprocedure of a motion information display apparatus according to thesecond embodiment;

FIG. 12A is a diagram for describing superimposition display of movingimage information and motion information;

FIG. 12B is a diagram for describing the superimposition display of themoving image information and the motion information;

FIG. 13A is a diagram for describing synchronization playback of themotion information;

FIG. 13B is a diagram for describing the synchronization playback of themotion information;

FIG. 14 is a diagram for describing superimposition display;

FIG. 15 is a diagram for describing synchronization playback of afootprint moving image;

FIG. 16 is a diagram for describing synchronization playback of a graphmoving image;

FIG. 17 is a diagram for describing synchronization playback of a movingimage accompanying highlighting; and

FIG. 18 is a diagram for describing an example applied to a serviceproviding apparatus.

DETAILED DESCRIPTION

A motion information display apparatus according to an embodimentincludes obtaining circuitry, identifying circuitry, and displaycontrolling circuitry. The obtaining circuitry obtains a plurality ofpieces of moving image information, and motion information thatindicates a motion of a subject included in each of the moving imageinformation. The identifying circuitry identifies a frame correspondingto a timing of a predetermined motion, from each frame group included ineach of the moving image information, based on the motion information.The display controlling circuitry performs display control of theplurality of pieces of moving image information, using the framecorresponding to a timing of a predetermined motion.

Hereinafter, a motion information display apparatus and a programaccording to embodiments will be described with reference to thedrawings. Note that, the motion information display apparatus describedbelow may be used as a single body of a motion information displayapparatus, or may be used being incorporated in a system such as amedical record system or a rehabilitation department system, forexample.

First Embodiment

FIG. 1 is a block diagram illustrating a configuration example of amedical-use information system according to a first embodiment. Amedical-use information system 1 according to the first embodiment is asystem that assists rehabilitation performed in a medical institution, ahome, an office, and the like. Here, “rehabilitation” refers to atechnique or a method for enhancing potential of a patient with adisorder, a chronic disease, or a disease of old age, which requireslong lasting treatment, to recover and facilitate a life function and asocial function. An example of the technique or the method includesfunction training for recovering and facilitating the life function andthe social function. Here, examples of the function training includewalking training and training of joint movable range. A person to berehabilitated is written as “target person”. The target person is, forexample, a sick person, an injured person, an aged person, or a disabledperson. Further, a person who assists the target person whenrehabilitation is performed is written as “helper”. The helper is, forexample, a medical professional such as a doctor, a physical therapistor a nurse who engages in a medical institution, or a caregiver, afamily member, or a friend who cares for the target person at home.Further, the rehabilitation is abbreviated as “rehab”.

As illustrated in FIG. 1, the medical-use information system 1 accordingto the first embodiment includes motion information collecting circuitry10, a motion information processing apparatus 20, a medical-use imagediagnostic apparatus 30, a medical-use image storage apparatus 40, and amotion information display apparatus 100. The motion informationprocessing apparatus 20, the medical-use image diagnostic apparatus 30,the medical-use image storage apparatus 40, and the motion informationdisplay apparatus 100 are in a state of being able to directly orindirectly communicate with each other by an in-hospital LAN (local areanetwork) 2 installed in a hospital, for example. When a picturearchiving and communication system (PACS) is introduced in themedical-use information system 1, for example, the devices mutuallytransmit/receive medical-use image information and the like according toa digital imaging and communications in medicine (DICOM) standard. Notethat the motion information display apparatus 100 may also be calledmedical-use image display apparatus.

The motion information collecting circuitry 10 detects a motion of aperson, an object, or the like in a space where rehabilitation isperformed, and collects motion information that indicates the motion ofa person, an object, or the like, and moving image information of thecaptured person or object. Note that, as the motion informationcollecting circuitry 10, Kinect (registered trademark) is used, forexample.

FIG. 2 is a block diagram illustrating a detailed configuration exampleof the motion information collecting circuitry 10 according to the firstembodiment. As illustrated in FIG. 2, the motion information collectingcircuitry 10 is connected to the motion information processing apparatus20, and includes color image collecting circuitry 11, distance imagecollecting circuitry 12, sound recognizing circuitry 13, and motioninformation generating circuitry 14. Note that the configuration of themotion information collecting circuitry 10 illustrated in FIG. 2 is anexample, and an embodiment is not limited to the example.

The color image collecting circuitry 11 captures a subject such as aperson, an object in a space where rehabilitation is performed, andcollects color image information. For example, the color imagecollecting circuitry 11 detects light reflected on a subject surfacewith a light-receiving element, and converts visible light into anelectrical signal. Then, the color image collecting circuitry 11generates one frame of color image information corresponding to acaptured range by converting the electrical signal into digital data.This one frame of color image information includes, for example,captured time information, and information in which RGB (red, green, andblue) values are associated with pixels included in the one frame. Thecolor image collecting circuitry 11 captures a moving image of thecaptured range by generating a plurality of continued frames of colorimage information from sequentially detected visible light. Note thatthe color image information generated by the color image collectingcircuitry 11 may be output as a color image in which the RGB values ofthe pixels are arranged in a bitmap. Further, the color image collectingcircuitry 11 includes a complementary metal oxide semiconductor (CMOS)or a charge coupled device (CCD), as the light-receiving element, forexample.

The distance image collecting circuitry 12 captures a subject such as toperson, an object in a space where rehabilitation is performed, andcollects distance image information. For example, the distance imagecollecting circuitry 12 irradiates the neighborhood with infrared light,and detects a reflected wave that is an irradiation wave reflected onthe subject surface, with a light-receiving element. Then, the distanceimage collecting circuitry 12 obtains a distance between the subject andthe distance image collecting circuitry 12, based on a phase differencebetween the irradiation wave and the reflected wave, and a time from theirradiation to the detection, and generates one frame of distance imageinformation corresponding to the captured range. This one frame ofdistance image information includes, for example, the captured timeinformation, and information in which each pixel included in thecaptured range is associated with the distance between the subject andthe distance image collecting circuitry 12 corresponding to the pixel.The distance image collecting circuitry 12 captures a moving image ofthe captured range by generating a plurality of continued frames ofdistance image information from sequentially detected reflected waves.Note that the distance image information generated by the distance imagecollecting circuitry 12 may be output as a distance image in which lightand shade of colors according to the distances of the respective pixelsis arranged in bitmap. Further, the distance image collecting circuitry12 includes a CMOS or a CCD, as the light-receiving element, forexample. This light-receiving element may be commonly used as thelight-receiving element used in the color image collecting circuitry 11.Further, the unit of the distances calculated in the distance imagecollecting circuitry 12 is, for example, meter [m].

The sound recognizing circuitry 13 collects sounds in the neighborhood,and identifies a direction of a sound source and recognizes a sound. Thesound recognizing circuitry 13 includes a microphone array provided witha plurality of microphones, and performs beam forming. The beam formingis a technology to selectively collect a sound from a specificdirection. For example, the sound recognizing circuitry 13 identifiesthe direction of the sound source by the beam forming using themicrophone array. Further, the sound recognizing circuitry 13 recognizesa word from the collected sounds using a known sound recognitiontechnology. That is, the sound recognizing circuitry 13 generatesinformation in which the word recognized by the sound recognitiontechnology, a direction from which the word is brought out, and a timeat which the word is recognized are associated with each other, as asound recognition result.

The motion information generating circuitry 14 generates motioninformation that indicates a motion of a person, an object, or the like.The motion information generating circuitry 14 generates the motioninformation by capturing a motion (gesture) of a person as a series of aplurality of postures (pauses). To describe an outline, the motioninformation generating circuitry 14 first obtains coordinates of jointsthat form a skeleton of a human body from the distance image informationgenerated by the distance image collecting circuitry 12 by patternmatching using a human body pattern. The coordinates of the jointsobtained from the distance image information are values expressed in acoordinate system (hereinafter, called “distance image coordinatesystem”) of distance image. Therefore, the motion information generatingcircuitry 14 next converts the coordinates of the respective joints inthe distance image coordinate system into values expressed in acoordinate system (hereinafter, called “world coordinate system”) in athree-dimensional space where rehabilitation is performed. Thecoordinates of the respective joints expressed in the world coordinatesystem serve as one frame of skeleton information. Further, a pluralityof frames of the skeleton information is the motion information.Hereinafter, processing of the motion information generating circuitry14 according to the first embodiment will be specifically described.

FIGS. 3A to 3C are diagrams for describing processing of the motioninformation generating circuitry 14 according to the first embodiment.FIG. 3A illustrates an example of a distance image generated by thedistance image collecting circuitry 12. Note that, in FIG. 3A, an imageis expressed by line drawing for convenience of description. However, anactual distance image is an image or the like expressed by light andshade of a color according to a distance. In the distance image, eachpixel includes a three-dimensional value in which a “pixel position X”in a right and left direction of the distance image, a “pixel positionY” in an up and down direction of the distance image, and a “distance Z”between a subject corresponding to the pixel and the distance imagecollecting circuitry 12, are associated with one another. Hereinafter,the values of the coordinates in the distance image coordinate systemare written as three-dimensional values (X, Y, Z).

In the first embodiment, the motion information generating circuitry 14stores human body patterns corresponding to various postures bylearning, in advance. The motion information generating circuitry 14acquires generated distance image information of each frame every timethe distance image information is generated by the distance imagecollecting circuitry 12. Then, the motion information generatingcircuitry 14 performs the pattern matching using the human body pattern,for the acquired distance image information of each frame.

Here, the human body pattern will be described. FIG. 3B illustrates anexample of the human body pattern. In the first embodiment, the humanbody pattern is a pattern used in the pattern matching with the distanceimage information, and thus expressed in the distance image coordinatesystem and including information of a surface of a human body(hereinafter, called “human body surface”), similar to a personvisualized in the distance image. For example, the surface of a humanbody corresponds to a surface of a skin or clothes of the person.Further, as illustrated in FIG. 3B, the human body pattern includesinformation of joints that forms a skeleton of the human body. That is,in the human body pattern, relative positional relationship between thesurface of the human body and the respective joints is known.

In the example illustrated in FIG. 3B, the human body pattern includesinformation of 20 points of joints from a joint 3 a to joint 3 t. Amongthe joints, the joint 3 a corresponds to a head, a joint 3 b correspondsto a central portion of both shoulders, a joint 3 c corresponds to awaist, and a joint 3 d corresponds to a central portion of a hip.Further, a joint 3 e corresponds to a right shoulder, a joint 3 fcorresponds to a right elbow, a joint 3 g corresponds to a right wrist,and a joint 3 h corresponds to a right hand. Further, a joint 3 icorresponds to a left shoulder, a joint 3 j corresponds to a left elbow,a joint 3 k corresponds to a left wrist, and a joint 3 l corresponds toa left hand. Further, a joint 3 m corresponds to a right hip, a joint 3n corresponds to a right knee, a joint 3 o corresponds to a right ankle,and a joint 3 p corresponds to a tarsus of a right foot. Further, ajoint 3 q corresponds to a left hip, a joint 3 r corresponds to a leftknee, a joint 3 s corresponds to a left ankle, and a joint 3 tcorresponds to a tarsus of a left foot.

Note that, in FIG. 3B, the case where the human body pattern has theinformation of 20 points of joints has been described. However, anembodiment is not limited to the case, and positions and the number ofjoints may be arbitrarily set by an operator. For example, when onlychange of movements of the arms and feet is captured, the information ofthe joints 3 b and 3 c, of the joints 3 a to 3 d, may not be obtained.Further, when change of a movement of the right hand is captured indetail, not only the joint 3 h, but also a joint of a finger of theright hand may be newly set. Note that, the joints 3 a, 3 h, 3 l, 3 p,and 3 t of FIG. 3B are end portions of bones and thus are different fromso-called joints. However, they are important points expressingpositions and directions of bones. Therefore, here, they are describedas joints for convenience of description.

The motion information generating circuitry 14 performs the patternmatching with the distance image information of each frame using thehuman body pattern. For example, the motion information generatingcircuitry 14 extracts a person in a certain posture from the distanceimage information by performing pattern matching between the human bodysurface of the human body pattern illustrated in FIG. 3B, and thedistance image illustrated in FIG. 3A. In this way, the motioninformation generating circuitry 14 obtains coordinates on the humanbody surface of the person visualized in the distance image. Further, asdescribed above, the relative positional relationship between the humanbody surface and the respective joints are known in the human bodypattern. Therefore, the motion information generating circuitry 14calculates coordinates of the respective joints in the person from thecoordinates on the human body surface, of the person visualized in thedistance image. In this way, as illustrated in FIG. 3C, the motioninformation generating circuitry 14 acquires the coordinates of therespective joints that form skeleton of the human body from the distanceimage information. Note that the coordinates of the respective jointsobtained here are coordinates in a distance coordinate system.

Note that, when performing the pattern matching, the motion informationgenerating circuitry 14 may auxiliary use information that indicatespositional relationship among the joints. The information that indicatesthe positional relationship among the joints includes, for example,coupling relationship between joints (for example, “the joint 3 a andthe joint 3 b are coupled” or the like), the ranges of motion of therespective joints, and the like. A joint is a portion that couples twoor more bones. An angle made by bones is changed according to change ofa posture, and the range of motion differs according to a joint. Forexample, the range of motion is expressed by a maximum value, a minimumvalue, or the like of the angle made by bones coupled by each joint. Forexample, when learning the human body pattern, the motion informationgenerating circuitry 14 also learns the ranges of motion of therespective joints, and stores the ranges of motion in association withthe respective joints.

Following that, the motion information generating circuitry 14 convertsthe coordinates of the respective joints in the distance imagecoordinate system into values expressed in the world coordinate system.The world coordinate system is a coordinate system in athree-dimensional space where rehabilitation is performed, and is acoordinate system in which the position of the motion informationcollecting circuitry 10 is the origin, a horizontal direction is an xaxis, a vertical direction is a y axis, and a direction perpendicular toan xy plane is a z axis, for example. Note that values of coordinates inthe z axis direction may be called “depth”.

Processing to convert the coordinates from the distance image coordinatesystem into the world coordinate system will be described. In the firstembodiment, the motion information generating circuitry 14 stores aconversion formula for converting the coordinates from the distanceimage coordinate system to the world coordinate system. For example,this conversion formula outputs the coordinates in the world coordinatesystem using the coordinates in the distance image coordinate system,and an incident angle of reflection light corresponding to thecoordinates as inputs. For example, the motion information generatingcircuitry 14 inputs coordinates (X1, Y1, Z1) of a certain joint, and theincident angle of reflection light corresponding to the coordinates tothe conversion formula to convert the coordinates (X1, Y1, Z1) of thecertain joint into coordinates (x1, y1, z1) in the world coordinatesystem. Note that, since corresponding relationship between thecoordinates in the distance image coordinate system, and the incidentangle of reflection light is known, the motion information generatingcircuitry 14 can input the incident angle corresponding to thecoordinates (X1, Y1, Z1) to the conversion formula. Here, the case inwhich the motion information generating circuitry 14 converts thecoordinates in the distance image coordinate system into the coordinatesin the world coordinate system has been described. However, thecoordinates in the world coordinate system can be converted intocoordinates in the distance coordinate system.

Then, the motion information generating circuitry 14 generates skeletoninformation from the coordinates of the respective joints expressed inthe world coordinate system. FIG. 4 is a diagram illustrating an exampleof the skeleton information generated by the motion informationgenerating circuitry 14. The skeleton information of each frame includescaptured time information of the frame, and the coordinates of therespective joints. For example, as illustrated in FIG. 4, the motioninformation generating circuitry 14 generates the skeleton informationin which joint identification information and coordinate information areassociated with each other. Note that, in FIG. 4, illustration of thecaptured time information is omitted. The joint identificationinformation is identification information for identifying the joints,and is set in advance. For example, joint identification information “3a” corresponds to the head, and joint identification information “3 b”corresponds to the central portion of both shoulders. The same appliesto other pieces of the joint identification information, and each of thejoint identification information indicates each corresponding joint.Further, the coordinate information indicates the coordinates of therespective joints in each frame, in the world coordinate system.

In the first row of FIG. 4, the joint identification information “3 a”and the coordinate information “(x1, y1, z1)” are associated with eachother. That is, the skeleton information of FIG. 4 indicates that thehead exists in the position of the coordinates (x1, y1, z1) in a certainframe. Further, in the second row of FIG. 4, the joint identificationinformation “3 b” and coordinate information “(x2, y2, z2)” areassociated with each other. That is, the skeleton information of FIG. 4indicates that the central portion of both shoulders exists in theposition of the coordinates (x2, y2, z2) in the certain frame. Further,the same applies to other joints, and the skeleton information indicatesthat the respective joints exist in the positions expressed byrespective coordinates in the certain frame.

As described above, the motion information generating circuitry 14performs the pattern matching with respect to the distance imageinformation of each frame every time acquiring the distance imageinformation of each frame from the distance image collecting circuitry12, and converts the coordinates from the distance image coordinatesystem into the world coordinate system, thereby to generate theskeleton information of each frame. Then, the motion informationgenerating circuitry 14 transmits the generated skeleton information ofeach frame to the motion information processing apparatus 20, as motioninformation.

Note that processing of the motion information generating circuitry 14is not limited to the above-described technique. For example, in theabove description, the technique in which the motion informationgenerating circuitry 14 performs the pattern matching using the humanbody pattern has been described. However, a technique is not limited tothe embodiment. For example, a technique to perform the pattern matchingusing patterns of different portions in place of the human body patternor together with the human body pattern may be performed.

Further, in the above description, the technique in which the motioninformation generating circuitry 14 obtains the coordinates of therespective joints from the distance image information has beendescribed. However, an embodiment is not limited to the technique. Forexample, a technique in which the motion information generatingcircuitry 14 obtains the coordinates of the respective joints usingcolor image information together with the distance image information maybe used. In this case, for example, the motion information generatingcircuitry 14 performs the pattern matching using the human body patternand the color image information expressed in a coordinate system of acolor image, and obtains coordinates of the human body surface from thecolor image information. In the coordinate system of a color image doesnot include the information of the “distance Z” in the distance imagecoordinate system. Therefore, the motion information generatingcircuitry 14 obtains the information of the “distance Z” from thedistance image information, for example, and obtains the coordinates ofthe respective joints in the world coordinate system by calculationprocessing using these two pieces of information.

Further, the motion information generating circuitry 14 transmits thecolor image information of a plurality of frames generated by the colorimage collecting circuitry 11 to the motion information processingapparatus 20, as moving image information. Further, the motioninformation generating circuitry 14 appropriately transmits the distanceimage information generated by the distance image collecting circuitry12 and the sound recognition result output by the sound recognizingcircuitry 13 to the motion information processing apparatus 20, asneeded. Note that a pixel position of the color image information and apixel position of the distance image information can be associated witheach other in advance according to positions and capturing directions ofthe color image collecting circuitry 11 and the distance imagecollecting circuitry 12. Therefore, the pixel position of the colorimage information and the pixel position of the distance imageinformation can be associated with the world coordinate systemcalculated by the motion information generating circuitry 14. Further,the height or the length of each portion of the body (the length of thearm or the length of an abdomen), and a distance between two pixelsspecified on the color image can be obtained using the association and adistance [m] calculated by the distance image collecting circuitry 12.Further, similarly, captured time information of the color imageinformation and captured time information of the distance imageinformation can be associated with each other in advance. Further, themotion information generating circuitry 14 can refer to the soundrecognition result and the distance image information, and when there isthe joint 3 a in the vicinity of the direction from which a word isbrought out, a sound of which has been recognized at a certain time, themotion information generating circuitry 14 can output the word as a worduttered by the person that includes the joint 3 a. Further, the motioninformation generating circuitry 14 appropriately transmits theinformation that indicates positional relationship among the joints tothe motion information processing apparatus 20, as needed.

Note that the case in which a motion of a single person is detected bythe motion information collecting circuitry 10 has been described here.However, an embodiment is not limited to the case. The motioninformation collecting circuitry 10 may detect motions of a plurality ofpersons as long as the motions are included in the captured range of themotion information collecting circuitry 10. Note that, when a pluralityof persons is captured in the distance image information of the sameframe, the motion information collecting circuitry 10 associates theskeleton information of the plurality of persons generated from thedistance image information of the same frame with one another, andoutputs the associated information to the motion information processingapparatus 20, as the motion information.

Further, a configuration of the motion information collecting circuitry10 is not limited to the above-described configuration. For example,when generating the motion information by detecting a motion of a personby another motion capture, such as an optical system, a mechanicalsystem, or a magnetic system, the motion information collectingcircuitry 10 may not necessarily include the distance image collectingcircuitry 12. In such a case, the motion information collectingcircuitry 10 includes a marker to be mounted on the human body and asensor that detects the marker, as a motion sensor, in order to detectthe motion of the person. Then, the motion information collectingcircuitry 10 detects the motion of the person using the motion sensor,and generates the motion information. Further, the motion informationcollecting circuitry 10 associates the pixel position of the color imageinformation and the coordinates of the motion information using theposition of the marker included in the image captured by the color imagecollecting circuitry 11, and then appropriately outputs the associatedinformation to the motion information processing apparatus 20, asneeded. Further, for example, the motion information collectingcircuitry 10 may not include the sound recognizing circuitry 13 when notoutputting the sound recognition result to the motion informationprocessing apparatus 20.

Further, in the above-described embodiment, the motion informationcollecting circuitry 10 outputs the coordinates of the world coordinatesystem, as the skeleton information. However, the embodiment is notlimited thereto. For example, the motion information collectingcircuitry 10 may output the coordinates in the distance image coordinatesystem before conversion, and the conversion from the distance imagecoordinate system into the world coordinate system may be performed atthe side of the motion information processing apparatus 20, as needed.

Referring back to the description of FIG. 1, the motion informationprocessing apparatus 20 processes various types of information forassisting rehabilitation. The motion information processing apparatus 20is an information processing apparatus such as a computer, or aworkstation.

For example, the motion information processing apparatus 20 accepts themoving image information and the motion information transmitted by themotion information generating circuitry 14. Then, the motion informationprocessing apparatus 20 outputs the accepted moving image informationand motion information to a monitor, a speaker, and the like, andtransmits the accepted moving image information and motion informationto the medical-use image storage apparatus 40.

Further, for example, the motion information processing apparatus 20accepts the distance image information and the sound recognition resulttransmitted by the motion information generating circuitry 14. Then, themotion information processing apparatus 20 appropriately outputs theaccepted distance image information and sound recognition result to amonitor, a speaker, and the like, and appropriately transmits theaccepted distance image information and sound recognition result to themedical-use image storage apparatus 40.

The medical-use image diagnostic apparatus 30 is a apparatus such as anX-ray diagnostic apparatus, an X-ray computed tomography (CT) apparatus,a magnetic resonance imaging (MRI) apparatus, a ultrasonic diagnosticapparatus, a single photon emission computed tomography (SPECT)apparatus, a positron emission computed tomography (PET) apparatus, aSPECT-CT apparatus in which an SPECT apparatus and an X-ray CT apparatusare integrated, a PET-CT apparatus in which a PET apparatus and an X-rayCT apparatus are integrated, or a specimen inspection apparatus. Forexample, the medical-use image diagnostic apparatus 30 captures a testobject such as a patient according to an operation from a film operatorwho captures the test object such as a patient, and generatesmedical-use image information. Then, the medical-use image diagnosticapparatus 30 transmits the generated medical-use image information tothe medical-use image storage apparatus 40.

The medical-use image storage apparatus 40 stores the various types ofinformation for assisting rehabilitation. For example, the medical-useimage storage apparatus 40 includes a database in which the imageinformation is stored, and stores the various types of informationtransmitted by the motion information processing apparatus 20 and themedical-use image diagnostic apparatus 30, in the database.

For example, the medical-use image storage apparatus 40 stores themoving image information and the motion information transmitted by themotion information processing apparatus 20 in association with eachother, for each target person. The moving image information is the colorimage information of the plurality of frames generated by the colorimage collecting circuitry 11, and is information of a moving image of acaptured target person who undergoes rehabilitation, for example.Further, the moving image information includes information thatindicates a type of the rehabilitation (for example, walking training,training of joint movable range, or the like) undergone by the capturedtarget person, as incidental information. Further, the motioninformation is the skeleton information corresponding to each frame ofthe moving image information, and indicates a motion of the targetperson captured in the moving image information, for example.

The motion information display apparatus 100 displays the various typesof information for assisting rehabilitation. For example, the motioninformation display apparatus 100 obtains the moving image informationand the motion information from the medical-use image storage apparatus40, and displays the acquired information.

FIG. 5 is a block diagram illustrating a detailed configuration exampleof the motion information display apparatus 100 according to the firstembodiment. As illustrated in FIG. 5, the motion information displayapparatus 100 includes output circuitry 110, input circuitry 120,storage circuitry 130, and controlling circuitry 140.

Here, the motion information display apparatus 100 according to thefirst embodiment can appropriately display the moving image of capturedrehabilitation, by processing described below. For example, whenarranging and displaying a plurality of moving images of walkingtraining, the motion information display apparatus 100 can display themoving images, synchronizing timings of the motions.

For example, walking from standing upright is performed by repetition ofmotions below:

(1) Put the right foot forward.

(2) Allow the right foot to land on a floor surface.

(3) Shift the center of gravity of the boy toward the right foot, andputs the weight on the right foot.

(4) Remove the weight from the left foot.

(5) Hit the floor surface with the left foot.

(6) Put the left foot farther forward than the right foot.

(7) Allow the left foot to land on the floor surface.

(8) Shift the center of gravity of the body toward the left foot, andputs the weight on the left foot.

(9) Remove the weight from the right foot.

(10) Hit the floor surface with the right foot.

(11) Put the right foot farther forward than the left foot.

Hereinafter, returning to (2), and the motions from (2) to (11) arerepeated until the end of walking. Note that, in the walking, both feetcannot come off the ground at the same time, and putting the armforward, which is at the opposite side to the foot put forward,maintains the balance of the center of gravity of the body.

Here, for example, it can be considered that, when two moving images aredisplayed and compared, that is, when a moving image in which walking ofa healthy person is captured and a moving image in which walkingtraining of a target person is captured are displayed and compared, itis useful to display the moving images using predetermined timings. Asan example, it can be considered that, when timings at which the rightfeet land on the ground first time are caused to accord with each otherand the two moving images are played back at the same time, walkingspeeds of the healthy person and the target person walk can be compared.Further, as another example, it can be considered that, when landingtimings of the right feet are caused to accord with each other in thetwo moving images, walking forms of the healthy person and the targetperson can be compared. Therefore, the motion information displayapparatus 100 can display a plurality of moving images using the landingtimings of the right feet by processing described below, when displayingtwo moving images in which the walking training is captured.

Note that, in the first embodiment, a case of using a timing at whichthe right foot lands on the ground will be described. However, anembodiment is not limited to the case. For example, in the embodiment, atiming at which another motion is performed, such as a timing at whichthe left foot hits the floor surface, may be used. Further, anembodiment is not limited to the case where the moving image of thewalking training is displayed, and may be applied to a case where amoving image of another training such as the training of joint movablerange is displayed, for example. Further, an embodiment is not limitedto the case where the healthy person and the target person are compared,and for example, a current target person and a past target person may becompared. To be specific, in the embodiment, the target person before anoperation and the target person after the operation may be compared, orthe target person immediately after the operation and the target personafter elapse of several months after the operation may be compared.Further, in the embodiment, three or more moving images may be compared.Further, in the embodiment, a plurality of moving images may be arrangedand displayed in the horizontal direction or in the vertical direction.

The output circuitry 110 outputs the various types of information forassisting rehabilitation. For example, the output circuitry 110 displaysa graphical user interface (GUI) for allowing an operator who operatesthe motion information display apparatus 100 to input various requestsusing the input circuitry 120, displays an output image generated in themotion information display apparatus 100, and outputs a warning sound.For example, the output circuitry 110 is a monitor, a speaker, aheadphone, or a headphone portion of a head set.

The input circuitry 120 accepts an input of the various types ofinformation for assisting rehabilitation. For example, the inputcircuitry 120 accepts an input of various types of requests from theoperator of the motion information display apparatus 100, and transfersthe accepted various types of requests to the motion information displayapparatus 100. For example, the input circuitry 120 is a mouse, akeyboard, a touch command screen, a trackball, a microphone, amicrophone portion of a head set.

The storage circuitry 130 includes moving image information storagecircuitry 131 and motion information storage circuitry 132. For example,the storage circuitry 130 is a semiconductor memory element such as arandom access memory (RAM) or a flash memory, or a storage device suchas a hard disk device or an optical device.

The moving image information storage circuitry 131 stores the movingimage information to be displayed. The moving image information is thecolor image information of the plurality of frames, and is the movingimage information of the captured target person who undergoes therehabilitation, for example. Further, the moving image informationincludes the information that indicates a type of the rehabilitation(for example, walking training, training of joint movable range, or thelike) undergone by the captured target person, as the incidentalinformation. For example, the moving image information storage circuitry131 stores the moving image information obtained from the medical-useimage storage apparatus 40 by obtaining circuitry 141 described below.Note that, the moving image information stored in the moving imageinformation storage circuitry 131 may be acquired from the motioninformation processing apparatus 20.

The motion information storage circuitry 132 stores the motioninformation corresponding to the moving image information stored in themoving image information storage circuitry 131. The motion informationis the skeleton information corresponding to each frame of the movingimage information, and indicates, for example, a motion of the targetperson captured in the moving image information. For example, the motioninformation storage circuitry 132 stores the motion information obtainedfrom the medical-use image storage apparatus 40 by the obtainingcircuitry 141. Note that, the motion information stored in the motioninformation storage circuitry 132 may be acquired from the motioninformation processing apparatus 20.

The controlling circuitry 140 includes obtaining circuitry 141,identifying circuitry 142, and display controlling circuitry 143. Forexample, the controlling circuitry 140 can be realized by an integratedcircuit such as an application specific integrated circuit (ASIC) or afield programmable gate array (FPGA), or can be realized by execution ofa predetermined program by a central processing unit (CPU).

The obtaining circuitry 141 obtains a plurality of pieces of the movingimage information, and the motion information that indicates a motion ofa subject included in each of the moving image information.

For example, the obtaining circuitry 141 accesses the medical-use imagestorage apparatus 40, and accepts an input for specifying a targetperson ID that identifies the target person, from the operator throughthe input circuitry 120. Following that, the obtaining circuitry 141displays a list of the moving image information that can be played backabout the target person of the specified target person ID, of the movingimage information stored in the medical-use image storage apparatus 40to the output circuitry 110. Then, the obtaining circuitry 141 acceptsan input for selecting the moving image information to be displayed fromthe displayed list, from the operator through the input circuitry 120.Then, the obtaining circuitry 141 obtains the selected one piece of themoving image information, and the motion information corresponding tothe moving image information, from the medical-use image storageapparatus 40, and stores the acquired moving image information andmotion information in the moving image information storage circuitry 131and the motion information storage circuitry 132, respectively. Notethat the obtaining circuitry 141 repeats the above-described processingas needed, thereby to obtain the repeated number of pieces of the movingimage information and the motion information, as information to bedisplayed.

The identifying circuitry 142 identifies the frame corresponding to atiming of a predetermined motion performed in the rehabilitation, fromeach frame group included in each of the moving image information, basedon the motion information. For example, the identifying circuitry 142accepts an input for specifying the purpose of the rehabilitation, fromthe operator through the input circuitry 120. The identifying circuitry142 identifies the frame corresponding to the timing of thepredetermined motion according to the specified purpose of therehabilitation.

FIGS. 6A and 6B are diagrams for describing processing of theidentifying circuitry 142 according to the first embodiment. In FIGS. 6Aand 6B, a case in which the identifying circuitry 142 specifies a timingat which the right foot lands on the ground from the moving image of thewalking training will be described. Note that FIG. 6A is a graphillustrating change with time of a y coordinate of the tarsus of theright foot in the walking of the healthy person, and FIG. 6B is a graphillustrating change with time of a y coordinate of the tarsus of theright foot in the walking of the target person. In FIGS. 6A and 6B, thehorizontal axes represent time, and the vertical axes represent a valueof the y coordinate of the tarsus of the right foot. That is, y=0indicates that the tarsus of the right foot lands on the floor surface.

FIGS. 6A and 6B exemplarily illustrate the walking from standingupright. That is, ta1 of FIG. 6A corresponds to a timing at which thehealthy person puts the right foot forward. Further, ta2, ta4, and tabcorrespond to a timing at which the healthy person allows the right footto land on the floor surface. Further, ta3, ta5, and ta7 correspond to atiming at which the healthy person hits on the floor surface with theright foot. Further, tb1 of FIG. 6B corresponds to a timing at which thetarget person puts the right foot forward. Further, tb2, tb4, and tb6correspond to a timing at which the target person allows the right footto land on the floor surface. Further, tb3 and tb5 correspond to atiming at which the target person hits on the floor surface with theright foot. As illustrated in FIGS. 6A and 6B, the healthy person walksmore regularly and with a higher pace than the target person, while thetarget person walks irregularly and with a slower pace than the healthyperson. Note that, hereinafter, the timing at which the foot lands onthe ground may be abbreviated as “landing timing”.

Here, a case where improvement of the walking speed of the target personis specified as the purpose of the rehabilitation will be described. Inthis case, the identifying circuitry 142 accepts an input for specifyingthe improvement of the walking speed of the target person, from theoperator. When the improvement of the walking speed has been specified,the identifying circuitry 142 identifies a frame corresponding to thefirst landing timing of the right foot, from each frame group includedin each of the moving image information obtained by the obtainingcircuitry 141. To be specific, as illustrated in FIG. 6A, theidentifying circuitry 142 refers to the motion information of thehealthy person, and identifies the first frame from a plurality offrames in which the y coordinate of the tarsus (joint 3 p) of the rightfoot of the healthy person is 0 for a fixed period. In the exampleillustrated in FIG. 6A, the identifying circuitry 142 identifies framesrespectively corresponding to ta2, ta4, and ta6, as the landing timingsof the right foot of the healthy person. Then, the identifying circuitry142 identifies the frame corresponding to ta2 in the earliest time, fromthe identified frames respectively corresponding to ta2, ta4, and ta6,as the first landing timings of the right foot. Further, as illustratedin FIG. 6B, the identifying circuitry 142 identifies a framecorresponding to tb2, as the first landing timing of the right foot ofthe target person, by similar processing.

Following that, the identifying circuitry 142 identifies the size of theperson captured in each of the moving image information. For example,the identifying circuitry 142 identifies a ratio occupied by the personto the number of pixels of the color image in the up and down direction,in the frame corresponding to the identified landing timing, as the sizeof the person. To be specific, the identifying circuitry 142 obtains thecoordinates of the head (joint 3 a) and the tarsus of the right foot(joint 3 p) from the skeleton information of the frames respectivelycorresponding to ta2 and tb2. The identifying circuitry 142 calculatespixel positions respectively corresponding to the obtained coordinatesof the head and the tarsus of the right foot. The identifying circuitry142 calculates the ratio occupied by the number of pixels from the headto the tarsus of the right foot to the number of pixels of the colorimage in the up and down direction, as the size of the person. In thisway, the identifying circuitry 142 identifies the size of the personcaptured in each of the moving image information.

Further, a case where improvement of the walking form of the targetperson is specified as the purpose of rehabilitation will be described.In this case, the identifying circuitry 142 accepts an input forspecifying the improvement of the walking form of the target person,from the operator. When the improvement of the walking form has beenspecified, the identifying circuitry 142 identifies frames correspondingto all landing timings of the right foot, from each frame group includedin each of the moving image information obtained by the obtainingcircuitry 141. To be specific, as illustrated in FIG. 6A, theidentifying circuitry 142 refers to the motion information of thehealthy person, and identifies the first frame from a plurality offrames in which the y coordinate of the tarsus (joint 3 p) of the rightfoot of the healthy person is 0 for a fixed period. In the exampleillustrated in FIG. 6A, the identifying circuitry 142 identifies theframes respectively corresponding to ta2, ta4, and ta6, as the landingtimings of the right foot of the healthy person. Further, as illustratedin FIG. 6B, the identifying circuitry 142 identifies frames respectivelycorresponding to tb2, tb4, and tb6, as all landing timings of the rightfoot of the target person, by similar processing.

Following that, the identifying circuitry 142 identifies the size of theperson captured in each of the moving image information. For example,the identifying circuitry 142 identifies a ratio occupied by the personto the number of pixels of the color image in the up and down direction,in the frame corresponding to the identified landing timing, as the sizeof the person. Here, when there is a plurality of identified landingtimings, the identifying circuitry 142 identifies the size of the personusing the first landing timing. To be specific, the identifyingcircuitry 142 identifies the size of the person by processing similar tothe above-described processing.

Note that the purpose of rehabilitation is not limited to theabove-described examples, and is appropriately set according to the typeof the rehabilitation and a state of the target person. Then, processingfor identifying an appropriate timing according to the set purpose isset to the identifying circuitry 142 in advance.

A case where an effect of an operation is confirmed by how much thewalking speed of the target person is improved before and after theoperation will be described as an embodiment. In this case, theidentifying circuitry 142 accepts an input for specifying improvement ofthe walking speed of the target person, from the operator. When theimprovement of the walking speed has been specified, the identifyingcircuitry 142 identifies frames corresponding to the first landingtiming of the right foot, from respective frame groups included in themoving image information before the operation and the moving imageinformation after the operation.

The display controlling circuitry 143 performs display control of aplurality of pieces of the moving image information using framescorresponding to a timing of a predetermined motion. For example, whenarranging and displaying a plurality of moving images of the walkingtraining, the display controlling circuitry 143 performs the displaycontrol of a plurality of pieces of the moving image information usingthe frames corresponding to the landing timing of the right footidentified by the identifying circuitry 142.

FIG. 7 is a diagram for describing processing of the display controllingcircuitry 143 according to the first embodiment. FIG. 7 illustrates anexample of a display screen 7 a displayed in the output circuitry 110 bythe display controlling circuitry 143. The display screen 7 a includes aregion 7 b where the moving image information of the healthy person isdisplayed, a region 7 c where a graph that indicates change with time ofthe y coordinate of the tarsus of the right foot of the healthy personis displayed, a region 7 d where the moving image information of thetarget person is displayed, and a region 7 e where a graph thatindicates change with time of the y coordinate of the tarsus of theright foot of the target person is displayed. Further, the displayscreen 7 a includes a walking speed comparison playback button 7 f forspecifying the improvement of the walking speed of the target person asthe purpose of rehabilitation, and a walking motion comparison playbackbutton 7 g for specifying the improvement of the walking form of thetarget person.

Here, a case where the improvement of the walking speed of the targetperson is specified as the purpose of rehabilitation will be described.For example, the identifying circuitry 142 accepts information thatindicates that the walking speed comparison playback button 7 f has beenpressed by the operator, from the input circuitry 120. Then, the displaycontrolling circuitry 143 performs playback control so as to cause thesizes of the target person and start timings in the moving imageinformation to accord with each other. To be specific, the displaycontrolling circuitry 143 enlarges or reduces the moving imageinformation of the healthy person and the moving image information ofthe target person such that the healthy person and the target person canhave nearly the same size, using the sizes of the persons identified bythe identifying circuitry 142. Following that, the display controllingcircuitry 143 plays back these two pieces of moving image information atthe same time such that the moving image information of the healthyperson is started from the frame corresponding to ta2, and the movingimage information of the target person is started from the framecorresponding to tb2.

Further, for example, the case where an effect of an operation isconfirmed by how much the walking speed of the target person is improvedbefore and after the operation will be described. In this case, forexample, the identifying circuitry 142 accepts the information thatindicates that the walking speed comparison playback button 7 f has beenpressed by the operator, from the input circuitry 120. Then, the displaycontrolling circuitry 143 performs the playback control so as to causethe sizes of the target person and the start timings of the moving imageinformation before the operation and the moving image information afterthe operation to accord with each other.

Further, a case where the improvement of the walking form of the targetperson is specified as the purpose of rehabilitation will be described.For example, the display controlling circuitry 143 accepts informationthat indicates that the walking motion comparison playback button 7 ghas been pressed by the operator, from the input circuitry 120. Then,the display controlling circuitry 143 performs the playback control soas to cause the sizes of the target person and the landing timings inthe moving image information to accord with each other. To be specific,the display controlling circuitry 143 enlarges or reduces the movingimage information of the healthy person and the moving image informationof the target person such that the healthy person and the target personcan have nearly the same size, using the sizes of the persons identifiedby the identifying circuitry 142. Following that, the displaycontrolling circuitry 143 synchronizes the landing timings of thehealthy person with the landing timings of the target person, and playsback the moving image information. To be more specific, since the numberof frames from tb1 to tb2 is smaller than the number of frames from ta1to ta2, the display controlling circuitry 143 partially interpolatesframes from tb1 to tb2. Accordingly, the display controlling circuitry143 plays back the moving image information so as to display the framescorresponding to ta2 and tb2 at the same time. Further, since the numberof frames from tb2 to tb4 is larger than the number of frames from ta2to ta4, the display controlling circuitry 143 partially thins out theframes from tb2 to tb4. Accordingly, the display controlling circuitry143 plays back the moving image information so as to display the framescorresponding to ta4 and tb4 at the same time. As described above, thedisplay controlling circuitry 143 plays back the moving imageinformation so as to cause the respective landing timings to accord witheach other.

Note that a method of synchronizing and playing back the moving imagesby the display controlling circuitry 143 is not limited to theabove-described method. For example, when synchronizing ta2 and tb2, thedisplay controlling circuitry 143 may temporarily stop the moving imageinformation of the target person at the frame corresponding to tb2,which is played back first, until the frame corresponding to ta2 isplayed back in the moving image information of the healthy person.

Note that, in the above description, the case in which the start timingsto play back the moving images are caused to accord with each otherusing the first landing timings has been described. However, anembodiment is not limited to the example. For example, in an embodiment,the start timings can be caused to accord with each other using thesecond or third landing timings. In this case, the moving images areplayed back at a normal speed until the second or third landing timing.Further, in an embodiment, for example, the second landing timing in onemoving image and the third landing timing in the other moving image, oftwo moving images, may be caused to accord with each other.

Further, in the above description, the case in which all of playbacktimes of the moving images are synchronized has been described. However,an embodiment is not limited to the case. For example, only fixed timesof the playback times may be synchronized, and times before and afterthe fixed times may be displayed without being synchronized.

Next, a processing procedure of the motion information display apparatus100 according to the first embodiment will be described with referenceto FIG. 8. FIG. 8 is a flowchart for describing an example of aprocessing procedure of the motion information display apparatus 100according to the first embodiment.

As illustrated in FIG. 8, when a plurality of pieces of moving imageinformation has been selected by the operator (Yes at step S101), theobtaining circuitry 141 obtains the selected pieces of moving imageinformation, and corresponding motion information (step S102). Note thatthe obtaining circuitry 141 is in a stand-by state until the pluralityof pieces of moving image information is selected (No at step S101).

Following that, the identifying circuitry 142 accepts the informationthat indicates that the playback button has been pressed (step S103).When the walking speed comparison playback button has been pressed atstep S103, the identifying circuitry 142 identifies a framecorresponding to the landing timing of the right foot at a point oftime, based on each motion information obtained by the obtainingcircuitry 141 (step S104). Following that, the identifying circuitry 142identifies the size of the person captured in each of the moving imageinformation (step S105). Then, the display controlling circuitry 143performs the playback control so as to cause the sizes of the targetperson and the start timings of the moving image information to accordwith each other (step S106), and terminates the processing.

Further, when the walking motion comparison playback button has beenpressed at step S103, the identifying circuitry 142 identifies framescorresponding to all landing timings of the right foot, based on themoving information obtained by the obtaining circuitry 141 (step S107).Following that, the identifying circuitry 142 identifies the size of theperson captured in each of the moving image information (step S108).Then, the display controlling circuitry 143 performs the playbackcontrol so as to cause the sizes of the target person and the landingtimings of the moving image information to accord with each other (stepS109), and terminates the processing.

Note that a processing procedure is not limited to the above-describedprocessing procedure. For example, there is a case where the informationthat indicates the purpose of rehabilitation is stored in arehabilitation electronic medical record server in association with thetarget person ID. In such a case, the identifying circuitry 142 obtainsthe information that indicates the purpose of rehabilitation from therehabilitation electronic medical record server using the target personID. Then, the identifying circuitry 142 may automatically select andexecute the processing of step S104 or the processing of step S107 byusing the obtained information that indicates the purpose ofrehabilitation, even if the playback button is not pressed, which is theprocessing of step S103.

As described above, the motion information display apparatus 100according to the first embodiment obtains a plurality of pieces ofmoving image information, and motion information that indicates a motionof a subject included in each of the moving image information. Then, themotion information display apparatus 100 identifies a framecorresponding to a timing of a predetermined motion performed in therehabilitation, from each frame group included in each of the movingimage information, based on the motion information. Then, the motioninformation display apparatus 100 performs the display control of theplurality of pieces of moving image information using the framecorresponding to a timing of a predetermined motion. Therefore, themotion information display apparatus 100 can appropriately display themoving image of the captured rehabilitation.

For example, when displaying and comparing walking of the healthy personand walking of the target person, the motion information displayapparatus 100 according to the first embodiment causes timings at whichthe right feet land on the ground for the first time to approximatelyaccord with each other, and plays back the timings, thereby to displayand compare the walking speeds of the healthy person and the targetperson.

Further, for example, when displaying and comparing the walking of thehealthy person and the walking of the target person, the motioninformation display apparatus 100 according to the first embodimentcauses the timings at which the right feet land on the ground toapproximately accord with each other (synchronizes landing periods ofthe right feet), thereby to display and compare the walking forms of thehealthy person and the target person.

Further, for example, in evaluation of an outcome of the rehabilitationby a doctor or a physical therapist, the motion information displayapparatus 100 can compare the outcome by a display method according toan evaluation purpose.

Second Embodiment

In the first embodiment, the case in which the moving image of thewalking training is displayed has been described. However, an embodimentis not limited to the case. For example, an embodiment may be applied toa case where a moving image of another training, such as training ofjoint movable range, is displayed. Therefore, in a second embodiment, acase where a motion information display apparatus 100 displays a movingimage of training of joint movable range will be described.

Here, the training of joint movable range is training for enlarging amovable range of a joint. For example, the training of joint movablerange of a right elbow is to train how much a joint 3 f of FIG. 3B canbe moved. That is, the training of joint movable range of a right elbowis training to repeat a motion (bending and stretching) of operating theright elbow (the joint 3 f) without changing the position of an upperarm (from a joint 3 e to the joint 3 f) having a state in which theright arm (form the joint 3 e to a joint 3 g) is stretched straight, asa reference position, and bringing a right wrist (the joint 3 g) closeto a right shoulder (the joint 3 e). In the training of joint movablerange, an angle (the joint movable range) made by a front arm (from thejoint 3 f to the joint 3 g) before movement and the front arm after themovement is used as an index of evaluation, for example.

A motion information display apparatus 100 according to the secondembodiment has a configuration similar to the motion information displayapparatus 100 illustrated in FIG. 5, and a part of processing inidentifying circuitry 142 and display controlling circuitry 143 isdifferent. Therefore, in the second embodiment, the point different fromthe first embodiment will be mainly described, and points havingfunctions similar to the configurations described in the firstembodiment are denoted with the same reference signs as FIG. 5, anddescription is omitted. Note that, in the description below, a case inwhich moving image information of captured training of joint movablerange of a healthy person, and moving image information of capturedtraining of joint movable range of a target person are obtained byobtaining circuitry 141, as moving image information to be displayed,will be described.

For example, when the moving image information of captured training ofjoint movable range of a healthy person, and the moving imageinformation of captured training of joint movable range of a targetperson have been obtained by the obtaining circuitry 141, theidentifying circuitry 142 extracts information that indicates a type ofrehabilitation (training of joint movable range), from incidentalinformation of the moving image information. Accordingly, theidentifying circuitry 142 executes processing below.

FIGS. 9A and 9B are diagrams for describing processing of theidentifying circuitry 142 according to the second embodiment. In FIGS.9A and 9B, a case in which the identifying circuitry 142 identifies atiming at which the joint movable range is maximized, from a movingimage of the training of joint movable range, will be described. Notethat FIG. 9A is a graph illustrating change with time of the jointmovable range of the healthy person, and FIG. 9B is a graph illustratingchange with time of the joint movable range of the target person. InFIGS. 9A and 9B, the horizontal axes represent time, and vertical axesrepresent a value of the joint movable range. Note that this value isobtained such that the identifying circuitry 142 calculates the anglemade by the front arm (from the joint 3 f to the joint 3 g) before themovement and the front arm after the movement.

FIGS. 9A and 9B exemplarily illustrate the training of joint movablerange from a reference position. That is, tc1 of FIG. 9A corresponds toa timing of a state in which the healthy person stretches the right armstraight (reference position). Further, tc2, tc4, and tc6 correspond totimings at which the joint movable range of the healthy person ismaximized. Further, tc3, tc5, and tc7 correspond to timings at which thejoint movable range of the healthy person is minimized. Further, td1 ofFIG. 9B corresponds to a timing of a state in which the target personstretches the right arm straight (reference position). Further, td2 andtd4 correspond to timings at which the joint movable range of the targetperson is maximized. Further, tc3 and tc5 correspond to timings at whichthe joint movable range of the target person is minimized. Asillustrated in FIGS. 9A and 9B, the healthy person moves the right elbowat a higher pace than the target person, and the target person operatesthe right elbow at a slower pace than the healthy person. Note that,hereinafter, the timing at which the joint movable range is maximizedmay be abbreviated as “maximum timing”. Further, for example, from tc2to tc4 may be called “displacement period of an angle”.

Here, a case in which improvement of a motion speed of the target personis specified as the purpose of rehabilitation will be described. In thiscase, the identifying circuitry 142 accepts an input for specifying theimprovement of a motion speed of the target person, from the operator.When the improvement of a motion speed has been specified, theidentifying circuitry 142 identifies a frame corresponding to the firstmaximum timing, from each frame group included in each of the movingimage information obtained by the obtaining circuitry 141. To bespecific, as illustrated in FIG. 9A, the identifying circuitry 142refers to the motion information of the healthy person, and identifiesthe frame where a differential value of the joint movable range of theright elbow of the healthy person is changed from a positive value to anegative value. In the example illustrated in FIG. 9A, the identifyingcircuitry 142 identifies the frames respectively corresponding to tc2,tc4, and tc6, as the maximum timings of the healthy person. Then, theidentifying circuitry 142 identifies the frame corresponding to tc2,which is the earliest time, of the identified frames respectivelycorresponding to tc2, tc4, and tc6, as the first maximum timing.Further, as illustrated in FIG. 9B, the identifying circuitry 142identifies the frame corresponding to td2, as the first maximum timingof the target person, by similar processing.

Following that, the identifying circuitry 142 identifies the size of thesubject captured in each of the moving image information (step S205).For example, the identifying circuitry 142 identifies a ratio occupiedby the upper arm of the person to the number of pixels of a color imagein the right and left direction, in the frame corresponding to theidentified maximum timing, as the size of the subject. To be specific,the identifying circuitry 142 obtains coordinates of the right shoulder(joint 3 e) and the right elbow (joint 3 f) from the skeletoninformation of the frames respectively corresponding to tc2 and td2. Theidentifying circuitry 142 calculates pixel positions respectivelycorresponding to the obtained coordinates of the right shoulder and theright elbow. The identifying circuitry 142 calculates a ratio occupiedby the number of pixels from the right shoulder to the right elbow tothe number of pixels of the color image in the right and left direction,as the size of the subject. In this way, the identifying circuitry 142identifies the size of the subject captured in each of the moving imageinformation.

Further, a case in which improvement of a motion form of the targetperson is specified as the purpose of rehabilitation will be described.In this case, the identifying circuitry 142 accepts an input forspecifying the improvement of a motion form of the target person, fromthe operator. When the improvement of a motion form has been specified,the identifying circuitry 142 identifies frames corresponding to allmaximum timings, from each frame group included in each of the movingimage information obtained by the obtaining circuitry 141. To bespecific, as illustrated in FIG. 9A, the identifying circuitry 142refers to the motion information of the healthy person, and identifiesframes in which a differential value of the joint movable range of theright elbow of the healthy person is changed from a positive value to anegative value. In the example illustrated in FIG. 9A, the identifyingcircuitry 142 identifies frames respectively corresponding to tc2, tc4,and tc6, as the maximum timings of the healthy person. Further, asillustrated in FIG. 9B, the identifying circuitry 142 identifies framesrespectively corresponding to td2 and td4, as all maximum timings of thetarget person, by similar processing.

Following that, the identifying circuitry 142 identifies the size of thesubject captured in each of the moving image information. For example,the identifying circuitry 142 identifies a ratio occupied by the upperarm of the person to the number of pixels of the color image in theright and left direction, in the frame corresponding to the identifiedmaximum timing, as the size of the subject. Here, when there is aplurality of the identified maximum timings, the identifying circuitry142 identifies the size of the subject using the first maximum timing.

Note that the purpose of rehabilitation is not limited to theabove-described example, and is appropriately set according to a type ofthe rehabilitation or a state of the target person. Then, processing foridentifying an appropriate timing according to the set purpose is set tothe identifying circuitry 142 in advance.

A case in which, regarding an effect of an operation, improvement of amotion form of the target person is specified before and after theoperation will be described as an embodiment. In this case, theidentifying circuitry 142 accepts an input for specifying theimprovement of a motion form of the target person, from the operator.When the improvement of a motion form has been specified, theidentifying circuitry 142 identifies frames corresponding to all maximumtimings, from respective frame groups included in the moving imageinformation before the operation and the moving image information afterthe operation.

For example, the display controlling circuitry 143 performs displaycontrol of the plurality of pieces of moving image information accordingto the purpose of rehabilitation. For example, when arranging anddisplaying a plurality of moving images of the training of joint movablerange, the display controlling circuitry 143 performs the displaycontrol of the plurality of pieces of moving image information using theframes corresponding to the maximum timings identified by theidentifying circuitry 142.

FIG. 10 is a diagram for describing processing of the displaycontrolling circuitry 143 according to the second embodiment. FIG. 10illustrates an example of a display screen 10 a displayed by the displaycontrolling circuitry 143. The display screen 10 a includes a region 10b in which the moving image information of the healthy person isdisplayed, a region 10 c where a graph indicating change with time ofthe joint movable range of the healthy person is displayed, a region 10d in which the moving image information of the target person isdisplayed, and a region 10 e in which a graph indicating change withtime of the joint movable range of the target person is displayed.Further, the display screen 10 a includes a speed comparison playbackbutton 10 f for specifying improvement of a motion speed of the targetperson as the purpose of rehabilitation, and a motion comparisonplayback button 10 g for specifying the improvement of a motion form ofthe target person. Further, the display screen 10 a includes graphs 10 hand 10 i that display a bar of a current value (an angle of a currentlydisplayed frame in the case of moving image playback) and a pointindicating a maximum value (a maximum value of angles up to thecurrently displayed frame in the case of moving image playback), in thetraining of joint movable range.

Here, an example in which the improvement of a motion speed of thetarget person is specified as the purpose of rehabilitation will bedescribed. For example, the identifying circuitry 142 acceptsinformation that indicates that the speed comparison playback button 10f has been pressed by the operator, from an input circuitry 120. Then,the display controlling circuitry 143 performs playback control so as tocause the sizes of the target person and start timings of the movingimage information to accord with each other. To be specific, the displaycontrolling circuitry 143 enlarges or reduces the moving imageinformation of the healthy person and the moving image information ofthe target person such that the upper arms of the healthy person and thetarget person can be nearly the same size, using the sizes of thesubjects identified by the identifying circuitry 142. Following that,the display controlling circuitry 143 plays back these two pieces ofmoving image information at the same time such that the moving imageinformation of the healthy person is started from the framecorresponding to tc2, and the moving image information of the targetperson is started from the frame corresponding to td2.

Further, a case in which an effect of an operation is confirmed by howmuch a movable speed (motion speed) of the target person is improvedbefore and after the operation will be described. In this case, forexample, the identifying circuitry 142 accepts information thatindicates that the speed comparison playback button 10 f has beenpressed by the operator, from the input circuitry 120. Then, the displaycontrolling circuitry 143 performs the playback control so as to causethe sizes of the target person and start timings of the moving imageinformation before the operation and the moving image information afterthe operation to accord with each other. Note that the above-describedexample is a mere example, and there is a case in which the effect of anoperation before and after the operation is confirmed by how much themovable range (angle) is improved, for example.

Further, a case in which improvement of a motion form of the targetperson is specified as the purpose of rehabilitation will be described.For example, the identifying circuitry 142 accepts information thatindicates that the motion comparison playback button 10 g has beenpressed by the operator, from the input circuitry 120. Then, the displaycontrolling circuitry 143 performs the playback control so as to causethe sizes of the target person and the maximum timings of the movingimage information to accord with each other. To be specific, the displaycontrolling circuitry 143 enlarges or reduces the moving imageinformation of the healthy person and the moving image information ofthe target person such that the upper arms of the healthy person and thetarget person can have nearly the same size, using the sizes of thesubjects identified by the identifying circuitry 142. Following that,the display controlling circuitry 143 synchronizes the maximum timingsof the healthy person with the maximum timings of the target person, andplays back the moving image information. To be specific, since thenumber of frames from td1 to td2 is larger than the number of framesfrom tc1 to tc2, the display controlling circuitry 143 partially thinsout the frames from td1 to td2. Accordingly, the display controllingcircuitry 143 plays back the moving image information so as to displaythe frames corresponding to tc2 and td2 at the same time. Further, sincethe number of frames from td2 to td4 is larger than the number of framesfrom tc2 to tc4, the display controlling circuitry 143 partially thinsout the frames from td2 to td4. Accordingly, the display controllingcircuitry 143 plays back the moving image information so as to displaythe frames corresponding to tc4 and td4 at the same time. In this way,the display controlling circuitry 143 plays back the moving imageinformation so as to cause the maximum timings to accord with eachother.

Next, a processing procedure of the motion information display apparatus100 according to the second embodiment will be described with referenceto FIG. 11. FIG. 11 is a flowchart for describing an example of aprocessing procedure of the motion information display apparatus 100according to the second embodiment.

As illustrated in FIG. 11, when a plurality of pieces of moving imageinformation has been selected (Yes at step S201), the obtainingcircuitry 141 obtains the selected pieces of moving image information,and corresponding motion information (step S202). Note that theobtaining circuitry 141 is in a stand-by state until the plurality ofpieces of moving image information is selected (No at step S201).

Following that, the identifying circuitry 142 accepts the informationthat indicates that the playback button has been pressed (step S203).When the speed comparison playback button has been pressed at step S203,the identifying circuitry 142 identifies a frame corresponding to themaximum timing of a certain point of time, based on each of the motioninformation obtained in the obtaining circuitry 141 (step S204).Following that, the identifying circuitry 142 identifies the size of thesubject captured in each of the moving image information (step S205).Then, the display controlling circuitry 143 performs the playbackcontrol so as to cause the sizes of the target person and the starttimings of the moving image information to accord with each other (stepS206), and terminates the processing.

Further, when the motion comparison playback button has been pressed atstep S203, the identifying circuitry 142 identifies frames correspondingto all maximum timings, based on each of the motion information obtainedby the obtaining circuitry 141 (step S207). Following that theidentifying circuitry 142 identifies the size of the subject captured ineach of the moving image information (step S208). Then, the displaycontrolling circuitry 143 performs the playback control so as to causethe sizes of the target person and the maximum timings of the movingimage information to accord with each other (step S209), and terminatesthe processing.

As described above, the motion information display apparatus 100according to the second embodiment can be applied to the case where amoving image of another training is displayed, for example, the casewhere a moving image of the training of joint movable range isdisplayed.

Third Embodiment

The first and second embodiments have been described so far. However,various types of different embodiments may be implemented, other thanthe first and second embodiments.

(Superimposition Display of Moving Image Information and MotionInformation)

For example, in the first and second embodiments, the cases where two ormore pieces of moving image information are arranged and played backhave been described. However, an embodiment is not limited to the cases.For example, motion information may be superimposed and displayed on theplayed-back moving image information.

FIGS. 12A and 12B are diagrams for describing superimposition display ofmoving image information and motion information. FIG. 12A exemplarilyillustrates a display image in which corresponding motion information issuperimposed and displayed on moving image information of training ofjoint movable range of a healthy person. Further, FIG. 12B exemplarilyillustrates a display image in which corresponding motion information issuperimposed and displayed on moving image information of training ofjoint movable range of a target person.

As illustrated in FIGS. 12A and 12B, a display controlling circuitry 143superimposes and displays, on moving image information to be displayed,skeleton information corresponding to frames included in the movingimage information. To be specific, when displaying a moving image of thetraining of joint movable range of a right arm, the display controllingcircuitry 143 generates a display image in which information thatindicates positions of a right shoulder (a joint 3 e), a right elbow (ajoint 3 f), a right wrist (a joint 3 g), and a right hand (a joint 3 h)to be displayed are superimposed on a color image. Then, the displaycontrolling circuitry 143 performs display control of the generateddisplay image using a frame corresponding to a maximum timing identifiedby identifying circuitry 142. In this way, the motion informationdisplay apparatus 100 can clearly indicate the magnitude of the jointmovable range by superimposing and displaying the motion information onthe played-back moving image information.

(Synchronization Playback of Motion Information)

Further, for example, in an embodiment, not only the superimpositiondisplay, but also synchronization play back may be performed using onlythe motion information corresponding to the moving image information, asa moving image. To be specific, the display controlling circuitry 143synchronizes and displays, in place of the color image of the movingimage information to be displayed, the skeleton informationcorresponding to frames included in the moving image information, as amoving image.

FIGS. 13A and 13B are diagrams for describing the synchronizationplayback of the motion information. FIG. 13A exemplarily illustrates adisplay image that displays the motion information of the training ofjoint movable range of the healthy person, as a moving image. Further,FIG. 13B exemplarily illustrates a display image that displays themotion information of the training of joint movable range of the targetperson, as a moving image.

As illustrated in FIGS. 13A and 13B, the display controlling circuitry143 generates display images that display the skeleton informationcorresponding to frames of the moving information respectivelycorresponding to the moving image information of the healthy person andthe moving image information of the target person, as moving images. Tobe specific, when displaying the moving images of the training of jointmovable range of the right arm, the display controlling circuitry 143generates the display images that display information indicatingpositions of right shoulders (joints 3 e), right elbows (joints 3 f),right wrists (joints 3 g), and right hand (joints 3 h) to be displayed.Then, the display controlling circuitry 143 performs display control ofthe generated display images, using the frames corresponding to maximumtimings identified by the identifying circuitry 142. In this way, themotion information display apparatus 100 can clearly indicate themagnitude of the joint movable range by displaying only the motioninformation. Further, for example, the display controlling circuitry 143may display the joints, which are portions to be focused, in a differentcolor from others.

(Adjustment of Direction of Subject)

Further, for example, in the first and second embodiments, the cases ofdisplaying the moving images without causing directions of the subjectsto accord with each other have been described. However, it may bepossible to display the moving images after the directions of thesubjects are caused to accord with each other. For example, the displaycontrolling circuitry 143 performs display control after translating androtating at least one piece of the moving image information so as tocause the directions of the subjects to approximately accord with eachother between the moving image information.

FIGS. 13A and 13B are diagrams for describing adjustment of directionsof subjects. FIG. 13A exemplarily illustrates a display image thatdisplays the motion information of the training of joint movable rangeof the healthy person. Further, FIG. 13B exemplarily illustrates adisplay image that displays the motion information of the training ofjoint movable range of the target person.

As illustrated in FIGS. 13A and 13B, the display controlling circuitry143 displays the skeleton information corresponding to frames includedin the motion information corresponding to the moving image information,in place of the moving image information to be displayed, as movingimages, after adjusting the directions of the subjects. To be specific,when displaying moving images of the training of joint movable range ofthe right arm, the display controlling circuitry 143 extracts theinformation that indicates positions of right shoulders (joints 3 e),right elbows (joints 3 f), right wrists (joints 3 g), and right hands(joints 3 h) to be displayed, from the skeleton information of thehealthy person and the target person. Following that, the displaycontrolling circuitry 143 translates the motion information of thetarget person (calculates the coordinates) such that the position of theright elbow of the target person accord with the position of the rightelbow of the healthy person. Following that, the display controllingcircuitry 143 rotates the motion information of the target person aroundthe position of the right elbow (calculates the coordinated) such thatthe upper arm of the target person is superimposed on the upper arm ofthe healthy person. Following that, the display controlling circuitry143 rotates the motion information of the target person using theposition of the upper arm as a rotation axis (calculates thecoordinates) such that the direction into which the target person movesthe front arm accord with the direction into which the healthy personmoves the front arm. Then, the display controlling circuitry 143synchronizes and displays the motion information of the healthy personand the motion information of the target person, in the translated androtated direction. In this way, the motion information display apparatus100 can display the moving images after adjusting the directions of thesubjects. Note that a method of adjusting the directions of the subjectsis not limited to the above-described example, and three-dimensionalconversion or the like may be used, for example.

(Superimposition Display of Plurality of Moving Images)

Further, for example, in the first and second embodiments, the cases ofarranging and displaying the moving images to be displayed have beendescribed. However, an embodiment is not limited to the cases. Forexample, moving images to be displayed may be superimposed anddisplayed.

FIG. 14 is a diagram for describing superimposition display. FIG. 14exemplarily illustrates a display image in which the motion informationof the training of joint movable range of the healthy person, and themotion information of the training of joint movable range of the targetperson are superimposed and displayed.

As illustrated in FIG. 14, the display controlling circuitry 143superimposes and displays the skeleton information corresponding toframes included in the motion information to be displayed. To bespecific, when displaying moving images of the training of joint movablerange of the right arm, the display controlling circuitry 143 extractsthe information that indicates positions of right shoulders (joints 3e), right elbows (joints 3 f), right wrists (joints 3 g), and righthands (joints 3 h) to be displayed, from the skeleton information of thehealthy person and the target person. Then, the display controllingcircuitry 143 generates a superimposition image such that the positionsof the right shoulder and the right elbow of the healthy person, and thepositions of the right shoulder and the right elbow of the target personaccord with each other, respectively. Then, the display controllingcircuitry 143 displays the generated superimposition image in outputcircuitry 110. In this way, the motion information display apparatus 100can superimpose and display a plurality of moving images. At this time,types of lines or colors can be changed and displayed so that whether itis the motion information of the healthy person or the motioninformation of the target person can be distinguished.

(Synchronization Playback of Other Moving Images (Footprint MovingImages))

Further, moving images synchronized and played back by the motioninformation display apparatus 100 are not limited to the above-describedcolor image information and skeleton information. The motion informationdisplay apparatus 100 may synchronizes and plays back other movingimages based on the motion information. For example, the motioninformation display apparatus 100 may synchronizes and plays backfootprint moving images. The footprint moving image is an image thatsequentially displays landing positions where both feet of a person landon the floor surface when the person walks, for example.

FIG. 15 is a diagram for describing synchronization playback of thefootprint moving images. FIG. 15 illustrates an example of a displayscreen 7 a displayed in the output circuitry 110 by the displaycontrolling circuitry 143. The display screen 7 a is different from thedisplay screen 7 a exemplarily illustrated in FIG. 7, in that a region15 a is included in place of the region 7 c, and a region 15 b isincluded in place of the region 7 e. In the region 15 a, the footprintmoving image of the healthy person who is displayed in the moving imagein the region 7 b is displayed, and in the region 15 b, the footprintmoving image of the target person who is displayed in the moving imagein the region 7 d is displayed.

In the regions 15 a and 15 b, the horizontal direction corresponds to anx axis direction of a world coordinate system, and the verticaldirection corresponds to a z axis direction of the world coordinatesystem. That is, the regions 15 a and 15 b correspond to diagrams of afloor surface where a person walks, as viewed from above. Further, inthe region 15 a, the circles with numbers on non-colored backgroundcorrespond to the landing positions (coordinates) of the right foot ofthe person, the circles with numbers on colored background correspond tothe landing positions (coordinates) of the left foot of the person, andthe numbers in the circles indicate what number landing positions. Thisfootprint moving image is generated by the display controlling circuitry143, for example.

Here, processing of generating the footprint moving image by the displaycontrolling circuitry 143 will be described. For example, the displaycontrolling circuitry 143 generates the footprint moving image usinglanding timings identified by the identifying circuitry 142. To bespecific, the display controlling circuitry 143 obtains the coordinatesof the tarsi of the landed feet (the joint 3 p or the joint 3 t),respectively, from frames corresponding to the landing timings of theboth feet identified by the identifying circuitry 142. Then, the displaycontrolling circuitry 143 generates a moving image that displays xzcoordinates of the obtained respective tarsi on the floor surface, inthe frames corresponding to the landing timings of the feet of theperson. In the example illustrated in FIG. 15, the display controllingcircuitry 143 generates a moving image that displays the circle with “1”on non-colored background at the first landing timing of the right foot,the circle with “2” on colored background at the first landing timing ofthe left foot, the circle with “3” on non-colored background at thesecond landing timing of the right foot, the circle with “4” on coloredbackground at the second landing timing of the left foot, the circlewith “5” on non-colored background at the third landing timing of theright foot, and the circle with “6” on colored background at the thirdlanding timing of the left foot. In this way, the display controllingcircuitry 143 generates the footprint moving images respectivelydisplayed in the regions 15 a and 15 b.

As illustrated in FIG. 15, the display controlling circuitry 143performs display control of the footprint moving images respectivelydisplayed in the regions 15 a and 15 b using the frames corresponding tothe landing timings identified by the identifying circuitry 142. Forexample, when comparing walking speeds, the display controllingcircuitry 143 plays back these two footprint moving images at the sametime such that the first landing timings of the right feet displayed inthe regions 15 a and 15 b accord with each other. In this way, themotion information display apparatus 100 can perform playback control ofa plurality of footprint moving images.

(Synchronization Playback of Other Moving Images (Graph Moving Images))

Further, for example, the motion information display apparatus 100 maysynchronize and play back graph moving images, as other moving imagesbased on the motion information. The graph moving image is a graph thatillustrates information related to movement of a position of a targetposition, and is a moving image that plots a graph with elapsed time,the graph illustrating a predetermined value that is changed with amotion of a person. In other words, the graph moving image is a movingimage that displays a graph as if to draw a certain graph. Note that, asthe predetermined value, a value that can be calculated from theskeleton information, such as a walking speed of each frame or an angleof a knee joint of each frame, is arbitrarily selected by an operator.

FIG. 16 is a diagram for describing the synchronization playback of thegraph moving images. FIG. 16 illustrates an example of a display screen7 a displayed in the output circuitry 110 by the display controllingcircuitry 143. The display screen 7 a is different from the displayscreen 7 a exemplarily illustrated in FIG. 7, in that a region 16 a isincluded in place of the region 7 c, and a region 16 b is included inplace of the region 7 e. In the region 16 a, the graph moving image thatillustrates the walking speed of the healthy person who is displayed inthe moving image in the region 7 b is displayed, and in the region 15 b,the graph moving image that displays the walking speed of the targetperson who is displayed in the moving image in the region 7 d isdisplayed.

In the regions 16 a and 16 b, the horizontal direction corresponds totime, and the vertical direction corresponds to velocity (the walkingspeed). These footprint moving images are generated by the displaycontrolling circuitry 143, for example. To be specific, the displaycontrolling circuitry 143 calculates moving distances per unit time ofthe waists (joints 3 c) of respective persons displayed in the regions 7b and 7 d, for each frame, as the walking speeds. Then, the displaycontrolling circuitry 143 generates the graph moving images that plotthe calculated walking speeds of each frame, with elapsed time.

As illustrated in FIG. 16, the display controlling circuitry 143performs the display control of the graph moving images respectivelydisplayed in the regions 16 a and 16 b using the frames corresponding tothe landing timings identified by the identifying circuitry 142. Forexample, when comparing the walking speeds, the display controllingcircuitry 143 plays back these two footprint moving images at the sametime such that the first landing timings of the right feet displayed inthe regions 16 a and 16 b accord with each other. In this way, themotion information display apparatus 100 can perform playback control ofa plurality of graph moving images. Note that the graph moving imagesthat the motion information display apparatus 100 performs the playbackcontrol is not limited to the above-described example. For example, themotion information display apparatus 100 may perform the playbackcontrol of an arbitrary graph with values on the vertical axis, thevalues being able to be calculated from the skeleton information. To bespecific, the motion information display apparatus 100 may perform theplayback control of the graphs 10 h and 10 i exemplarily illustrated inFIG. 10.

Note that, in the example of FIG. 16, the horizontal direction is time,and the vertical direction is a speed. However, an embodiment is notlimited to the example. For example, when the horizontal direction istime, the vertical direction may be a position (coordinates), or may beacceleration. That is, an embodiment may be applied to a graph thatillustrates an arbitrary parameter that can be calculated from themotion information.

(Synchronization Playback of Other Moving Images (Moving ImagesAccompanying Highlighting))

Further, for example, the motion information display apparatus 100 maysynchronize and play back moving images accompanying highlighting of thedisplay screen 7 a illustrated in FIG. 7. As an example, the displaycontrolling circuitry 143 further displays information related toplayback control, of moving images based on the motion information.

FIG. 17 is a diagram for describing synchronization playback of movingimages accompanying highlighting. FIG. 17 illustrates an example of adisplay screen 7 a displayed in the output circuitry 110 by the displaycontrolling circuitry 143. This display screen 7 a is different from thedisplay screen 7 a exemplarily illustrated in FIG. 7, in that the joints3 p and 3 t are included in the regions 7 b and 7 d, and arrows thatindicate timings corresponding to frame used for synchronization(playback control) are included.

For example, the display controlling circuitry 143 highlights anddisplays points that indicate the joints 3 p and 3 t, which serve asfocused portions in walking training, in the regions 7 b and 7 d.Further, for example, the display controlling circuitry 143 highlightsand displays the arrows (the arrows with “synchronization” in thedrawing) that point out the timings of ta2, ta4, ta6, tb2, tb4, and tb6,which are frames used for synchronization, in regions 7 c and 7 e.

As illustrated in FIG. 17, the display controlling circuitry 143performs display control of moving images accompanying highlighting,using frames corresponding to landing timings identified by theidentifying circuitry 142. Note that, the display controlling circuitry143 may perform characteristic display for illustrating a timingcorresponding to a frame identified by the identifying circuitry 142.

(Application to Service Providing Apparatus)

In the first and second embodiments, the cases in which the motioninformation display apparatus 100 performs comparison display have beendescribed. However, an embodiment is not limited to the cases, and forexample, processing may be executed by a service providing apparatus ona network.

FIG. 18 is a diagram for describing an example applied to a serviceproviding apparatus. As illustrated in FIG. 18, a service providingapparatus 200 is arranged in a service center, and is connected withterminal apparatuses 300 arranged in a medical institution, a home, andan office, respectively, through a network 5, for example. Respectivemotion information collecting circuitry 10 are connected to the terminalapparatuses 300 arranged in the medical institution, the home, and theoffice, respectively. Further, each of the terminal apparatuses 300includes a client function that uses a service provided by the serviceproviding apparatus 200. Further, as the network 5, any type ofcommunication network regardless of wired or wireless, such as theInternet or a wide area network (WAN), can be employed.

For example, the service providing apparatus 200 has a similar functionto the motion information display apparatus 100 described in FIG. 5, andprovides the function to the terminal apparatus 300, as a service. Inthis case, the terminal apparatus 300 uploads, to the service providingapparatus 200, motion information and moving image information collectedby the motion information collecting circuitry 10 in the medicalinstitution, the home, or the office where rehabilitation is performed.The service providing apparatus 200 stores the motion information andthe moving image information uploaded by the terminal apparatus 300.

Here, the service providing apparatus 200 has function circuitryrespectively similar to the obtaining circuitry 141, the identifyingcircuitry 142, and the display controlling circuitry 143. Then, thefunction circuitry similar to the obtaining circuitry 141 obtains aplurality of pieces of the moving image information, and the motioninformation that indicates a motion of a subject included in each of themoving image information. Then, the function circuitry similar to theidentifying circuitry 142 identifies a frame corresponding to a timingof a predetermined motion performed in the rehabilitation, from eachframe group included in each of the moving image information, based onthe motion information. Then, the function circuitry similar to thedisplay controlling circuitry 143 performs display control of theplurality of the pieces of the moving image information using the framecorresponding to a timing of a predetermined motion.

To be specific, an operator specifies a plurality of pieces of themoving image information to be displayed, using the terminal apparatus300. The terminal apparatus 300 transmits information that indicates thespecified plurality of pieces of the moving image information to theservice providing apparatus 200. The service providing apparatus 200generates a display image for comparing and displaying the specifiedplurality of pieces of the moving image information by theabove-described function, and transmits the display image to theterminal apparatus 300. Accordingly, the operator can view the displayimage that displays and compares the specified plurality of pieces ofthe moving image information, on the terminal apparatus 300.

Note that the service providing apparatus 200 may have a configurationto perform a service of storing the motion information and the movingimage information only. In this case, the terminal apparatus 300 has afunction similar to the motion information display apparatus 100described in FIG. 5. Then, the terminal apparatus 300 appropriatelydownloads the moving image information and the motion information to bedisplayed, from the service providing apparatus 200, and performs theabove-described display control by functions respectively similar to theobtaining circuitry 141, the identifying circuitry 142, and the displaycontrolling circuitry 143.

OTHER EMBODIMENTS

The configuration of the motion information display apparatus 100 in theabove-described embodiments is a mere example, and integration andseparation of the circuitry can be appropriately performed.

For example, in the above-described embodiments, the cases applied tothe walking training and the training of joint movable range have beendescribed. However, an embodiment is not limited to the cases, and canbe applied to muscle build-up training of kinesitherapy, physiotherapy,or electrotherapy, for example.

Further, for example, the configuration of the motion informationdisplay apparatus 100 in the embodiments may be included in anyapparatus in the medical-use information system. For example, a motioninformation processing apparatus 20 may include a configuration of amotion information display apparatus 100.

Further, for example, in the above-described embodiments, the caseswhere the sizes of the subjects included in the moving image informationare caused to accord with each other have been described. However, anembodiment is not limited to the cases, and the processing to cause thesizes of the subjects to accord with each other may not be executed.

Further, for example, in the above-described embodiments, the cases inwhich the moving image information or the motion information isdisplayed have been described. However, an embodiment is not limited tothe cases. That is, the processing to display the moving imageinformation or the motion information may not be executed. For example,a motion information display apparatus 100 may identify a framecorresponding to a timing of a predetermined motion performed inrehabilitation, and further causes moving image information to accompanyinformation that indicates the identified frame, or display theinformation that indicates the identified frame in a graph or the like.When causing the moving image information to accompany the informationthat indicates the identified frame, for example, a apparatus differentfrom a motion information display apparatus 100 may obtain the movingimage information and incidental information, and perform displaycontrol like the above-described embodiments, according to theincidental information.

Further, for example, in the above-described embodiments, the caseswhere the moving image information stored in the medical-use imagestorage apparatus 40 is displayed and compared have been described.However, an embodiment is not limited to the cases. For example, amotion information display apparatus 100 may display and compare movingimage information acquired from a motion information processingapparatus 20 in approximately real time, and moving image informationstored in a medical-use image storage apparatus 40. According to thisconfiguration, the motion information display apparatus 100 cansynchronize timing of the moving image information stored in the pastand a motion about a target person, and display and compare theinformation and the motion, while capturing a moving image ofrehabilitation of the target person.

Further, the functions of the obtaining circuitry 141, the identifyingcircuitry 142, and the display controlling circuitry 143 described inthe above-described embodiments can be realized by software. Forexample, a computer executes a motion information display program thatdefines the procedure of the processing, which has been described beingperformed by the obtaining circuitry 141, the identifying circuitry 142,and the display controlling circuitry 143 in the above-describedembodiments, so that the functions of the obtaining circuitry 141, theidentifying circuitry 142, and the display controlling circuitry 143 arerealized. This motion information display program is stored in a harddisk or a semiconductor memory element, for example, and is read andexecuted by a processor such as a CPU or a micro-processing circuitry(MPU). Further, this medical-use image display program can be recordedon a computer-readable recording medium such as a compact disc-read onlymemory (CD-ROM), a magnetic optical disk (MO), or a digital versatiledisk (DVD), and can be distributed.

Note that not only the rule information of rehabilitation, recommendedhelping state, and the like defined by Japanese Orthopaedic Associationaccording to the first to third embodiments, but also rules andregulations defined by other various organizations may be used. Forexample, various rules and regulations defined by “International Societyof Orthopaedic Surgery and Traumatology (SICOT)”, “American Academy ofOrthopaedic Surgeons (AAOS)”, “European Orthopaedic Research Society(EORS)”, “International Society of Physical and Rehabilitation Medicine(ISPRM)”, “American Academy of Physical Medicine and Rehabilitation(AAPM & R)”, or the like may be used.

According to at least one embodiment described above, the motioninformation display apparatus and the program of the present embodimentcan appropriately display a moving image of captured rehabilitation.

While certain embodiments have been described, these embodiments havebeen presented by way of example only, and are not intended to limit thescope of the inventions. Indeed, the novel embodiments described hereinmay be embodied in a variety of other forms; furthermore, variousomissions, substitutions and changes in the form of the embodimentsdescribed herein may be made without departing from the spirit of theinventions. The accompanying claims and their equivalents are intendedto cover such forms or modifications as would fall within the scope andspirit of the inventions.

What is claimed is:
 1. A motion information display apparatuscomprising: processing circuitry configured to obtain a first movingimage of a first subject moving in a periodic manner, first motioninformation that indicates chronological change of a position of acharacteristic point corresponding to a joint of the first subject inthe first moving image, a second moving image of a second subject movingin a periodic manner, and second motion information that indicateschronological change of a position of a characteristic pointcorresponding to a joint of the second subject in the second movingimage; retrieve, from storage, definition information for defining acharacteristic motion; identify, by analyzing the first moving imagebased on the first motion information, a first frame corresponding to atiming when the first subject performed the characteristic motioncorresponding to the definition information, from each frame groupincluded in the first moving image; identify, by analyzing the secondmoving image based on the second motion information, a second framecorresponding to a timing when the second subject performed thecharacteristic motion corresponding to the definition information fromeach frame group included in the second moving image; cause play back ofthe first moving image and the second moving image to have respectivetimings of the play back that accord with each other, using the firstframe and the second frame; display a first graph showing thechronological change in the position of the characteristic point of thefirst subject, based on the first motion information; display, on thefirst graph, a first marker showing a timing of motions corresponding tothe first frame; display a second graph showing the chronological changein the position of the characteristic point of the second subject, basedon the second motion information; and display, on the second graph, asecond marker showing a timing of motions corresponding to the secondframe.
 2. The motion information display apparatus according to claim 1,wherein the processing circuitry is further configured to obtaincoordinates of the characteristic point in the first subject, as thefirst motion information, obtain coordinates of the characteristic pointin the second subject, as the second motion information, identify thefirst frame based on the coordinates of the characteristic point in thefirst subject, and identify the second frame based on the coordinates ofthe characteristic point in the second subject.
 3. The motioninformation display apparatus according to claim 1, wherein theprocessing circuitry is further configured to identify the first frameand the second frame, according to at least one of a type ofrehabilitation and a purpose of rehabilitation.
 4. The motioninformation display apparatus according to claim 1, wherein theprocessing circuitry is further configured to cause the play back of thefirst moving image and the second moving image to accord with each otherby using parallel display or superimposition display to cause thetimings of the playback to approximately accord with each other betweenthe first moving image and the second moving image, using the firstframe and the second frame.
 5. The motion information display apparatusaccording to claim 1, wherein the processing circuitry is furtherconfigured to identify a size of the subject included in each of thefirst and second motion information, and cause the play back of thefirst moving image and the second moving image to accord with each otherby causing the sizes of the first subject and the second subject toapproximately accord with each other between the first and second movingimages, using the sizes of the first subject and the second subject. 6.The motion information display apparatus according to claim 1, whereinthe processing circuitry is further configured to cause the play back ofthe first moving image and the second moving image after translatingand/or rotating at least one of the first moving image and the secondmoving image to cause directions of the first subject and the secondsubject to accord with each other between the first moving image and thesecond moving image.
 7. The motion information display apparatusaccording to claim 1, wherein when causing the play back of the firstmoving image and the second moving image, the processing circuitry isfurther configured to display first skeleton information of the firstsubject included in the first moving image and second skeletoninformation of the second subject included in the second moving image.8. A motion information display method comprising: obtaining a firstmoving image of a first subject moving in a periodic manner, and firstmotion information that indicates chronological change of a position ofa characteristic point corresponding to a joint of the first subject inthe first moving image, a second moving image of a second subject movingin a periodic manner, and second motion information that indicateschronological change of a position of a characteristic pointcorresponding to a joint of the second subject in the second movingimage; retrieving, from storage, definition information for defining acharacteristic motion; identifying, by analyzing the first moving imagebased on the first motion information, a first frame corresponding to atiming when the first subject performed the characteristic motioncorresponding to the definition information, from each frame groupincluded in the first moving image; identifying, by analyzing the secondmoving image based on the second motion information, a second framecorresponding to a timing when the second subject performed thecharacteristic motion corresponding to the definition information, fromeach frame group included in the second moving image; causing play backof the first moving image and the second moving image to have respectivetimings of the play back that accord with each other, using the firstframe and the second frame; displaying a first graph showing thechronological change in the position of the characteristic point of thefirst subject, based on the first motion information; displaying, on thefirst graph, a first marker showing a timing of motions corresponding tothe first frame; displaying a second graph showing the chronologicalchange in the position of the characteristic point of the secondsubject, based on the second motion information; and displaying, on thesecond graph, a second marker showing a timing of motions correspondingto the second frame.
 9. The motion information display apparatusaccording to claim 1, wherein the first motion and the second motion aredefined by at least one of positional relationship between a pluralityof characteristic points of a human body and chronological change ofposition of a characteristic point of the human body.